In the area of business intelligence (BI), analytical queries of data repositories such as data warehouses process large amounts of data. Therefore, depending on the nature of the query, associated runtimes for processing such queries may range from a few seconds to several hours. In order to facilitate informed decisions regarding queries, estimates of expected resource consumption (usually expressed as runtime) associated with the query may be provided prior to initiation
In some database management systems, query optimizers may be utilized to calculate/estimate runtime using a cost model as well as statistical information pertaining to the queried data. For each proposed query, multiple execution plans, each having an associated cost estimation, are calculated. The query optimizer then chooses the cheapest execution plan according to the estimated costs.
Conventional query optimizers do not always provide precise runtime estimates as the underlying cost models are often oversimplified. Some optimizers utilize assumptions (e.g., uniform distribution of values, default statistics for certain tables) for which no statistical information has been gathered. Other optimizers focus solely on projected input/output (I/O) costs, thereby neglecting CPU or communication costs. For purposes of choosing an optimal or semi-optimal query execution plan, this latter arrangement may be acceptable as (a) non-I/O-related costs are likely to be roughly proportional to I/O costs and (b) costs are only considered in relation to each other (e.g., estimated costs for plan A are compared to estimated costs for plan B). However, in some cases, a more precise runtime estimate incorporating a wider range of cost factors (e.g. I/O, CPU, communication) and not only one dominating factor is required.
Most conventional query optimizers generate runtime estimates using Historic statistical information. Such assumptions do not always provide an accurate estimate of runtime which can result in unexpected costs associated with processing a query.